Machine learning (ML) has become an essential tool in Computational Fluid Dynamics (CFD) for modeling complex non-linear input-output relationships, often speeding up or replacing traditional numerical simulations. However, flow field classification, namely, inferring labels that cannot be computed from explicit equations, has been poorly investigated. Existing methods often rely on extracting expert-driven features from limited portions of the flow field, which can miss important phenomena and reduce model generality. We propose a novel methodology that leverages all available CFD data while preserving the underlying physical information. By employing physics-informed clustering, each simulation is reduced into clusters that represent distinct physical behaviors. These clusters are then analyzed using deep learning techniques, such as PointNet++, which is designed to handle point cloud data and capture spatial relationships between the clusters. Our approach significantly reduces computational costs while maintaining high accuracy, offering a more holistic and detailed understanding of flow fields compared to traditional methods. This scalable solution is not only applicable to aerodynamic simulations but also has the potential to extend to other areas of fluid dynamics.